Image processing apparatus, imaging apparatus, image processing method, and image processing program

ABSTRACT

An image processing apparatus including a processor is provided. The processor inputs, from an imaging element in which first imaging pixels having a lower SNR and second imaging pixels having a higher SNR are arranged on a same layer, a first captured image by the first imaging pixels and a second captured image by the second imaging pixels when the first imaging pixels and the second imaging pixels perform imaging simultaneously, selects a target pixel from the first captured image, extracts, from the second captured image or an interpolated image of the second captured image, pixels having luminance values close to a luminance value of a pixel corresponding to the target pixel in the second captured image or interpolated image, selects pixels corresponding to the extracted pixels from the first captured image, and corrects a luminance value of the target pixel based on luminance values of the selected pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.16/659,065, filed on Oct. 21, 2019, which claims priority to and thebenefit of Japanese Patent Application No. 2018-202927 filed in theJapan Patent Office on Oct. 29, 2018, the entire contents of which areincorporated herein by reference.

BACKGROUND (a) Field

The described technology generally relates to an image processingapparatus, an imaging apparatus, an image processing method, and/or anon-transitory computer readable medium storing an image processingprogram.

(b) Description of the Related Art

An imaging apparatus such as a digital camera or a smartphone mayperform imaging in a dark scene. For example, an information processingapparatus such as the smartphone, may capture an infrared light imageand a visible light image, recognize a face image from the infraredlight image, extract an image for each part from the face image, correctthe image for each part using the visible light image and the infraredlight image in accordance with the extracted part, and combine thecorrected part images to reconstruct the image.

In the imaging apparatus, there is a great demand for high sensitivitydue to a demand for natural imaging without using a flash in the darkscene. However, considering the performance of the imaging apparatus, itis desirable to remove noise when perform imaging with increasedsensitivity in the dark.

SUMMARY

An example embodiment provides an image processing apparatus, an imagingapparatus, an image processing method, and/or a non-transitory computerreadable medium storing an image processing program, for acquiring animage excellent in color reproducibility and color resolution byappropriately removing noise from a noisy image captured in the dark.

According to an example embodiment, an image processing apparatus mayinclude processing circuitry configured to, receive a first capturedimage and a second captured image simultaneously captured via firstimaging pixels and second imaging pixels, respectively, that are one ofstacked or are arranged on a same layer of an imaging element, thesecond imaging pixels having a higher signal-to-noise ratio (SNR) thanthe first imaging pixels, select a target pixel from the first capturedimage, the target pixel in the first captured image having acorresponding pixel in the second captured image or an interpolatedimage of the second captured image, extract pixels from the secondcaptured image or the interpolated image of the second captured image togenerate extracted pixels such that first luminance values of theextracted pixels are close to a second luminance value of thecorresponding pixel, select pixels corresponding to the extracted pixelsfrom the first captured image as selected pixels, and correct aluminance value of the target pixel based on luminance values of theselected pixels.

According to another example embodiment, an image processing method mayinclude receiving a first captured image and a second captured imagecaptured simultaneously via first imaging pixels and second imagingpixels, respectively, stacked or are arranged on a same layer of animaging element, the second imaging pixels having a highersignal-to-noise ratio (SNR) than the first imaging pixels; selecting atarget pixel from the first captured image, the target pixel in thefirst captured image having a corresponding pixel corresponding in thesecond captured image or an interpolated image of the second capturedimage; extracting pixels from the second captured image or theinterpolated image of the second captured image to generate extractedpixels such that first luminance values of the extracted pixels areclose to a second luminance value of the corresponding pixel; selectingpixels corresponding to the extracted pixels from the first capturedimage as selected pixels; and correcting a luminance value of the targetpixel based on luminance values of the selected pixels.

According to yet another example embodiment, a non-transitory computerreadable medium may store an image processing program that, whenexecuted by a computing device, configures the computing device toreceive a first captured image and a second captured image bysimultaneously captured via first imaging pixels and second imagingpixels, respectively, that are one of stacked or are arranged on a samelayer of an imaging element, the second imaging pixels having a highersignal-to-noise ratio (SNR) than the first imaging pixels; select atarget pixel from the first captured image, the target pixel in thefirst captured image having a corresponding pixel corresponding theretoin the second captured image or an interpolated image of the secondcaptured image; extract pixels from the second captured image or theinterpolated image of the second captured image to generate extractedpixels such that the first luminance values of the extracted pixels areclose to a second luminance value of the corresponding pixel; selectingpixels corresponding to the extracted pixels from the first capturedimage as selected pixels; and correct a luminance value of the targetpixel based on luminance values of the selected pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for explaining an image processing method accordingto a comparative example.

FIG. 2 is a block diagram showing a schematic configuration of animaging apparatus 1 according to a first example embodiment.

FIG. 3 is a flowchart showing an image processing method according to afirst example embodiment.

FIG. 4 is a diagram for explaining interpolation processing according toa first example embodiment.

FIG. 5 is an example of a captured image and its interpolated imageaccording to a first example embodiment.

FIG. 6 is a diagram for explaining an image processing method accordingto a first example embodiment.

FIG. 7 is an example of a noise-removed image according to a firstexample embodiment and a comparative example.

FIG. 8 is a diagram for explaining a schematic configuration of animaging element according to a second example embodiment.

FIG. 9 is a diagram showing a relationship between a color temperature,a wavelength and a radiant energy.

FIG. 10 is a diagram for schematically explaining an image processingmethod according to a third embodiment.

FIG. 11 is a diagram for explaining a schematic configuration of animaging element according to a third example embodiment.

FIG. 12 is a flowchart showing an image processing method according to athird example embodiment.

FIG. 13 is a diagram for explaining an image processing method accordingto a third example embodiment.

FIG. 14 is a diagram for explaining a schematic configuration of animaging element according to a modification of a third exampleembodiment.

DETAILED DESCRIPTION

In the following detailed description, only certain example embodimentshave been shown and described, simply by way of illustration. As thoseskilled in the art would realize, the described example embodiments maybe modified in various different ways, all without departing from thespirit or scope thereof. Accordingly, the drawings and description areto be regarded as illustrative in nature and not restrictive. Likereference numerals designate like elements throughout the specification.

Hereinafter, an image processing apparatus and an image processingmethod according to example embodiments are described. Before describingthe example embodiments, as a comparative example, an image processingmethod of removing noise by correcting a luminance value of a targetpixel using luminance values of peripheral pixels is described.

Comparative Example

FIG. 1 is a diagram for explaining an image processing method accordingto a comparative example.

In an imaging element, imaging pixels including photoelectric conversionunits, which perform imaging in a red wavelength band, a greenwavelength band and a blue wavelength band (hereinafter referred to as“R imaging pixel”, “G imaging pixel” and “B imaging pixel”,respectively) are, for example, regularly arranged like a Bayer array.The image processing method shown in FIG. 1 sequentially select a pixelfrom which noise is to be removed (hereinafter referred to as “targetpixel”) in an image captured by the imaging element and corrects thetarget pixel.

In operation S51, the image processing method selects the target pixelfrom the captured image.

Next, in operation S52, the image processing method sets a luminancerange ±A determined by a magnitude of a luminance value (signal value)Ctr of the target pixel. For example, if the luminance value Ctr issmall, the luminance range ±A is set to be narrow. If the luminancevalue Ctr is great, the luminance range ±A is set to be wide.

Next, in operation S53, the image processing method searches for andextracts pixels whose luminance values fall within a range of thefollowing Equation 1 from pixels around the target pixel (hereinafterreferred to as “peripheral pixels”). For example, if the target pixel isa pixel imaged in the red wavelength band (hereinafter referred to as an“R pixel”), the image processing method searches for and extracts Rpixels whose luminance values fall within a range of the followingEquation 1 from pixels such as ±8 pixels or ±10 pixels in vertical andhorizontal directions around the target pixel (hereinafter referred toas “peripheral pixels”). As a result, pixels similar to the target pixelcan be extracted. Hereinafter, a pixel imaged in the green wavelengthband and blue wavelength band are also referred to as “G pixel” and “Bpixel”, respectively.

Ctr−A≤luminance values of peripheral pixels≤Ctr+A  Equation 1

In operation S54, the image processing method then calculates an averagevalue Ave of the luminance values of the extracted R pixels, and, inoperation S55, calculates an output value Output using the followingequation 2, and replaces the luminance value Ctr of the target pixelwith the output value Output thereby removing the noise of the targetpixel.

Output=Ctr+(Ave−Ctr)×gain  Equation 2

Here, gain≤1.0

In general, although the most effective way to reduce noise of an imageis to use an average value of a plurality of pixels as a value of thetarget pixel, if the average value of peripheral pixels is simply used,the image itself becomes blurred. Therefore, as described above, themethod of searching for the pixels similar to the target pixel based onfeatures such as an edge and a gradient, that is, the method ofsearching for a region having the same color and luminance distributionand using pixels in the region is performed. However, when searching forthe pixels similar to the target pixel, a ratio of noise to a signalincreases in a scene where the noise is most desired to be removed, forexample, in a dark scene so that it is difficult to distinguish thepixel similar to the target pixel and the pixel that is not similar tothe target pixel.

In the image processing method of the comparative example describedabove, the pixel search for calculating the average value Ave isperformed on pixels having the same color as the target pixel. Thisworks well for pixels with high signal intensity, that is, pixels with asomewhat high SNR (signal-to-noise ratio) (for example, R pixels in thedark or in a dark area). Therefore, the image processing method cansearch for pixels in the similar region like searching for the same edgeportion when the target pixel is the edge portion and searching for thesame background portion when the target pixel is the background portion.

However, in pixels with a low SNR (for example, G pixels and B pixels inthe dark or in the dark area), the signal is buried by the noise so thatit is difficult to search for pixels in the similar region and noiseremoval performance is reduced. As a result, there is a trade-offbetween (1) a situation where a corrected image becomes blurred when aluminance range ±A is widened to increase the noise removal effect and(2) a situation where the noise removal effect decreases when theluminance range ±A is narrowed to prevent the corrected image from beingblurred. Therefore, the image can be blurred or the noise cannot besufficiently reduced in the part where the noise is desired to bereduced.

Therefore, an image processing apparatus and an image processing methodaccording to various example embodiments, which can acquire an imageexcellent in color reproducibility and color resolution by appropriatelyremoving noise from a noisy image captured in the dark, will bedescribed below.

First Embodiment

An image processing apparatus and an image processing method accordingto a first example embodiment remove noise from an image captured by animaging element in which RGB imaging pixels are arranged on the samelayer. In this case, the first example embodiment selects a G pixel or Bpixel having a low SNR in the dark or in a dark part as a target pixel,extracts peripheral R pixels based on a luminance value of an R pixelcorresponding to the target pixel and having a high SNR in the dark orin the dark part, selects G pixels or B pixels corresponding to theextracted R pixels, and corrects the target pixel based on luminancevalues of the selected G pixels or B pixels.

Hereinafter, the image processing apparatus and the image processingmethod according to the first embodiment are described with reference tothe drawings.

First, a configuration of an image processing apparatus according to thefirst example embodiment is described.

The image processing apparatus according to the first example embodimentis an imaging apparatus, and particularly functions as an imageprocessing unit and removes the noise from the captured image.

FIG. 2 is a block diagram showing a schematic configuration of animaging apparatus 1 according to the first example embodiment.

Referring to FIG. 2, the imaging apparatus 1 is, for example, a digitalcamera, and includes a lens optical system 11, an imaging element 12, ananalog front end (AFE) circuit 13, an image processing unit 14, an imagedisplay unit 15, and an image storage unit 16.

The lens optical system 11 includes a lens, an aperture, and a shutter,and forms a subject image on an imaging surface of the imaging element12.

The imaging element 12 is an image sensor such as a charged coupleddevice (CCD) or a complementary metal oxide semiconductor (CMOS), andphotoelectrically converts the subject image to generate and output animage signal (RGB color signal) of the captured image. In the imagingelement 12, RGB imaging pixels may be arranged in a Bayer array on asame layer.

The AFE circuit 13 performs A/D (analog to digital) conversion on ananalog image signal that is output from the imaging element 12 and thensubjected to signal processing by a CDS circuit (not shown), and outputsa digital image signal to the image processing unit 14.

The image processing unit 14 calculates R signals at a G pixel positionand a B pixel position of the captured image by interpolation, andgenerates R signals for all the pixels.

Further, the image processing unit 14 sequentially selects the G pixelor the B pixel as a target pixel, extracts peripheral R pixels whoseluminance values are close to a luminance value of an R pixelcorresponding to the target pixel, selects G pixels or B pixelscorresponding to the extracted R pixels and corrects the target pixelbased on luminance values of the selected G pixels or B pixels.

Details of the processing of the image processing unit 14 will bedescribed later.

The image display unit 15 displays the captured image from which noisehas been removed, and the image recording unit 16 stores the capturedimage from which noise has been removed.

Each component realized by the image processing unit 14 may be realized,for example, by executing a program under a control of a processor (notshown) provided in the image processing unit 14 that is a computer. Forexample, in some example embodiments, the image processing unit 14 maybe implemented using processing circuitry such as hardware includinglogic circuits, a hardware/software combination such as a processorexecuting software; or a combination thereof. For example, theprocessing circuitry may include, but is not limited to, a centralprocessing unit (CPU), an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC) a programmable logic unit, a microprocessor, or anapplication-specific integrated circuit (ASIC), etc. The processingcircuitry may be configured as a special purpose computer to select atarget pixel (e.g., a G or B pixel) from a first captured image having arelatively low signal-to-noise ratio (SNR), extract peripheral pixels(e.g., R pixels) from a second captured image or an interpolated versionthereof having a relatively high SNR, select pixels (e.g., G or B)corresponding to the extracted pixels from the first captured imagehaving the low SNR, and correct a luminance value of the target pixel(e.g., the G or B pixel) in the first captured image based on luminancevalues of the selected pixels (e.g., G or B). Therefore, the processingcircuitry can acquire an image excellent in color reproducibility andcolor resolution by appropriately removing noise from a noisy imagecaptured in the dark.

More specifically, the image processing unit 14 may be realized byloading a program stored in a storage device (not shown) into a mainmemory (not shown) and executing the program under the control of theprocessor. The processor may be, for example, a processing unit such asa graphic processing unit (GPU), a central processing unit (CPU), amicroprocessor unit (MPU), or a micro controller unit (MCU). Eachcomponent is not limited to being realized by software by a program, andmay be realized by any combination of hardware, firmware, and software.

The above-described program may be supplied to the image processing unit14 with being stored in various types of non-transitory computerreadable media. The non-transitory computer readable media includevarious types of tangible storage media.

Examples of the non-transitory computer-readable media include amagnetic recording medium (for example, a flexible disk, a magnetictape, or a hard disk drive), a magneto-optical recording medium (forexample, a magneto-optical disk), a CD-ROM (Read Only Memory), a CD-R, aCD-R/W, and a semiconductor memory (for example, a mask ROM, a PROM(Programmable ROM), an EPROM (Erasable PROM), a flash ROM, or a RAM(random access memory)).

Further, the program may be supplied to the image processing unit 14 byvarious types of temporary computer readable media. Examples of thetransitory computer readable media include an electrical signal, anoptical signal, and an electromagnetic wave. The temporarycomputer-readable media may supply the program to the image processingunit 14 via a wired communication path such as an electric wire or anoptical fiber, or a wireless communication path.

Next, an operation, i.e., an image processing method of the imagingapparatus 1 according to the first example embodiment is described.

FIG. 3 is a flowchart showing an image processing method according tothe first example embodiment.

In operation S10, when a shutter of an imaging apparatus 1 is pressed,an imaging element 12 captures an image. Then, in operation S20, animage processing unit 14 calculates an R signal at a G pixel positionand a B pixel position of a captured image by interpolation processingusing peripheral R pixels, and generates an R interpolated image inwhich all pixels are R pixels.

FIG. 4 is a diagram for explaining interpolation processing according tothe first example embodiment. In FIG. 4, a left figure shows an enlargedpart of the captured image, and RGB pixels are arranged in the Bayerarray. A right figure shows an enlarged part of the R interpolatedimage, and G pixels and B pixels of the captured image are replaced withR pixels so that all the pixels are R pixels.

In this specification, pixels at the same position (coordinates) in twoimages are expressed as “corresponding to” each other. For example, inFIG. 4, a B pixel P1 of the captured image and an R pixel P2 of theinterpolated image are expressed as “corresponding to” as long as theyare at the same position.

FIG. 5 is an example of a captured image and its interpolated imageaccording to the first example embodiment. In FIG. 5, an upper row showsthe captured image, and a lower row shows the R interpolated image andreference G and B interpolated images for reference from the left.

The captured image is a night view of an urban area taken obliquely fromabove. In the captured image, a highway is placed on the left and rightslightly below the center of the image, multiple tall buildings standabove the highway, light leaks from several windows of the tallbuilding, and a general road is diagonally placed between the tallbuildings. In the captured image, the highway and the general road arereddish, and the high-rise building is appeared in gray or ocher.

Further, by reflecting such colors of the captured image, the Rinterpolated image has a high signal intensity and a high SNR, and the Gand B interpolated images (particularly, the B interpolated image) havea low signal intensity and a low SNR.

FIG. 6 is a diagram for explaining an image processing method accordingto the first embodiment.

Referring to FIGS. 3 and 6, in operation S30 of FIG. 3, whichcorresponds to operation S1 of FIG. 6, the image processing unit 14selects a G pixel or a B pixel as a target pixel in the captured image.In the following, a case where the B pixel is selected as the targetpixel is described, but the same processing is performed when the Gpixel is selected as the target pixel.

Next, in operation S40 of FIG. 3, which corresponds to operation S2 ofFIG. 6, the image processing unit 14 selects an R pixel in the Rinterpolated image corresponding to the B pixel. For example, referringto FIG. 4, when the B pixel P1 of the captured image is selected as thetarget pixel in step S30, the R pixel P2 of the R interpolated image isselected as the corresponding pixel in step S40.

Subsequently, in operation S50 of FIG. 3, which corresponds to operationS3 of FIG. 6, the image processing unit 14 sets a luminance range ±Adetermined by a magnitude of a luminance value Ctr of the R pixelselected in step S40.

Next, in operation S60 of FIG. 3, which corresponds to operation S4 ofFIG. 6, the image processing unit 14 searches for and extracts R pixelsin the R interpolated image whose luminance values falls within therange of the following equation 3 from peripheral pixels of the R pixelselected in step S40.

Ctr−A≤Luminance value of peripheral R pixel≤Ctr+A  Equation 3

In operation S70 of FIG. 3, which corresponds to operation S5 of FIG. 6,the image processing unit 14 may select B pixels in the captured imagecorresponding to the R pixels extracted in step S60. In the capturedimage, the pixels corresponding to the R pixels extracted in step S60may be R pixels or G pixels, but only the B pixels having the same coloras the target pixel are selected.

Next, in operation S80 of FIG. 3, which corresponds to operation S6 ofFIG. 6, the image processing unit 14 may calculate an average value Aveof luminance values of the B pixels selected in step S70 and calculatean output value Output based on the calculated average value Ave usingthe following equation 4. In operation S90 of FIG. 3, which correspondsto operation S7 of FIG. 6, the image processing unit 14 may replace theluminance value Ctr of the B pixel that is the target pixel with theoutput value Output so that the noise of the target pixel is removed.

Output=Ctr+(Ave−Ctr)×Gain  Equation 4

Here, Gain≤1.0

After the image processing unit 14 selects and corrects each of the Bpixel and G pixel in the captured image as the target pixel, and removesthe noise, in operation S100, an image display unit 15 displays thecaptured image from which the noise has been removed, and an imagestorage unit 16 records the captured image from which the noise has beenremoved.

FIG. 7 is an example of a noise-removed image according to the firstexample embodiment and the comparative example. FIG. 7 shows that thenoise has been removed from the captured image shown in FIG. 5, a leftfigure shows the noise-removed image according to the comparativeexample, and a right figure shows the noise-removed image according tothe first example embodiment.

When the noise-removed image according to the comparative example andthe noise-removed image according to the first embodiment are compared,edges of a window (frame) of each tall building, lighting fixtures onthe right side of the image and vehicles on the road are clearer in thenoise-removed image according to the first example embodiment.Therefore, an image excellent in color reproducibility and colorresolution can be acquired by appropriately removing the noise inaccordance with the image processing apparatus and the image processingmethod according to the first example embodiment.

Various modifications can be made in the image processing apparatus orthe image processing method according to the first example embodiment.For example, although removing the noise from the R pixels having thehigh SNR in the captured image is not particularly described in thefirst example embodiment, the noise may not be removed from the Rpixels, or the noise may be removed by using the method described in thecomparative example or a well-known method.

Further, although only the R interpolated image in which all pixels areR pixels is generated at step S20 in the first example embodiment, a Gor B interpolated image in which all pixels are G pixels or B pixels maybe also generated. In this case, when the B pixels (or G pixels)corresponding to the R pixels extracted in step S60 are selected in stepS70, the corresponding B pixels (or G pixels) always exist so that theaverage value Ave can be calculated by using more B pixels (or G pixels)in step S80.

Furthermore, in the first example embodiment, the G and B pixels arecorrected based on the R pixels assuming an image in which the red coloris dominant in the dark or in the dark part. However, in an image, suchas an image in which the subject is a daytime sky, in which the bluecolor is dominant, the R and G pixels with the high SNR may be correctedbased on the B pixels with the high SNR. In an image, such as an imagein which the subject is a daytime forest, in which the green color isdominant, the R and B pixels with the low SNR may be corrected based onthe G pixels with the high SNR. That is, the pixels having the low SNRmay be corrected based on the pixels having the high SNR.

In addition, in the first example embodiment, the B pixels correspondingto the extracted R pixels are selected in step S70, the average valueAve of the luminance values of the selected B pixels is calculated instep S80, the output value Output is calculated by using Equation 4, andthe luminance value Ctr of the target pixel B pixel is replaced with theoutput value Output so that the noise of the target pixel is removed.However, the method of removing the noise of the B pixel that is thetarget pixel by using the B pixels selected in step S70 is not limitedthereto.

Further, in the first example embodiment, the imaging apparatus 1 is notlimited to a digital camera, and may be various information processingdevices including an imaging element, such as a smartphone.

Furthermore, in the first example embodiment, the image processingapparatus functions as the imaging apparatus, in particular, the imageprocessing unit 14 to remove the noise from the captured image. However,the image processing apparatus may be provided separately from theimaging apparatus and remove the noise from the captured image output bythe imaging apparatus. For example, the image processing apparatus maybe a monitoring system that removes the noise from the captured imageoutput by a monitoring camera.

As described above, the image processing apparatus 1 according to thefirst example embodiment includes an image processing unit 14 thatinputs, from an imaging element 12 in which first imaging pixels havinga low SNR and second imaging pixels having a high SNR are arranged onthe same layer, a first captured image by the first imaging pixels and asecond captured image by the second imaging pixels when the firstimaging pixels and the second imaging pixels perform imagingsimultaneously, selects a target pixel from the first captured image,extracts pixels having luminance values, which are close to a luminancevalue of a pixel corresponding to the target pixel in an interpolatedimage of the second captured image, from the interpolated image, selectspixels corresponding to the extracted pixels from the first capturedimage, and corrects a luminance value of the target pixel based onluminance values of the selected pixels.

Accordingly, it is possible to acquire an image excellent in colorreproducibility and color resolution by appropriately removing noisefrom a noisy image captured in the dark.

In the image processing apparatus 1 according to the first exampleembodiment, the first imaging pixels may perform the imaging in a greenwavelength band or a blue wavelength band, and the second imaging pixelsmay perform the imaging in a red wavelength band.

Accordingly, G and B pixels having the low SNR can be corrected based onthe R pixels having the high SNR in the dark.

In the image processing apparatus 1 according to the first exampleembodiment, the image processing unit 14 may further select pixelscorresponding to the extracted pixels from an interpolated image of thefirst captured image.

Accordingly, the pixels corresponding to the extracted pixels can beselected from the first captured image and the interpolated image of thefirst captured image to correct the G and B pixels.

In addition, the imaging apparatus 1 according to the first exampleembodiment may include the imaging element 12 and the image processingunit 14.

Further, the image processing method according to the first embodimentincludes inputting, from an imaging element 12 in which first imagingpixels having a low SNR and second imaging pixels having a high SNR arearranged on the same layer, a first captured image by the first imagingpixels and a second captured image by the second imaging pixels when thefirst imaging pixels and the second imaging pixels perform imagingsimultaneously (S10), selecting a target pixel from the first capturedimage (S30), extracts pixels having luminance values, which are close toa luminance value of a pixel corresponding to the target pixel in thesecond captured image or a pixel corresponding to the target pixel in aninterpolated image of the second captured image, from the interpolatedimage (S60), selecting pixels corresponding to the extracted pixels fromthe first captured image (S70), and correcting a luminance value of thetarget pixel based on luminance values of the selected pixels (S90).

Accordingly, it is possible to acquire an image excellent in colorreproducibility and color resolution by appropriately removing noisefrom a noisy image captured in the dark.

Second Embodiment

The image processing apparatus and the image processing method accordingto the first example embodiment remove the noise from the image capturedby the imaging element 12 in which RGB imaging pixels are arranged onthe same layer. An image processing apparatus and an image processingmethod according to a second example embodiment remove noise from animage captured by an imaging element in which RGB imaging pixels arestacked. In this case, the second example embodiment selects a G pixelor B pixel having a low SNR in the dark or in a dark part as a targetpixel, extracts peripheral R pixels based on a luminance value of an Rpixel corresponding to the target pixel and having a high SNR in thedark or in the dark part, selects G pixels or B pixels corresponding tothe extracted R pixels, and corrects the target pixel based on luminancevalues of the selected G pixels or B pixels, like the first exampleembodiment.

Hereinafter, the image processing apparatus and the image processingmethod according to the second example embodiment is described withreference to the drawings.

The image processing apparatus according to the second exampleembodiment is an imaging apparatus, and particularly functions as animage processing unit and removes the noise from the captured image. Aschematic configuration of the imaging apparatus may be the same as thataccording to the first example embodiment shown in FIG. 2, andillustration and description thereof are omitted herein.

FIG. 8 is a diagram for explaining a schematic configuration of animaging element 22 according to the second example embodiment. FIG. 8shows a cross section when an imaging element 22 is viewed from adirection perpendicular to a stacking direction of imaging pixels.

The imaging element 22 has a stacked structure using a well-knowntechnique, for example, in which a lower layer is formed of asemiconductor material such as a silicon substrate and an upper layer isformed of a semiconductor material or an organic film. In the lowerlayer, imaging pixels having the low SNR in an image to be captured,i.e., G imaging pixels and B imaging pixels are arranged in a checkeredpattern. In the upper layer, R imaging pixels having the high SNR arearranged on an entire surface so as to correspond to the G imagingpixels or B imaging pixels in the lower layer. That is, in the upperlayer, all the imaging pixels are the R imaging pixels.

In a captured image captured by such an imaging element 22, in order toobtain an R pixel having the high SNR and a G pixel or B pixel havingthe low SNR at the same position (coordinates) of the image and removenoise, the same processing as the image processing method according tothe first example embodiment may be performed. In this case, since Rpixels of all the pixels are obtained, the image processing unit 14according to the second example embodiment may not generate an Rinterpolated image in operation S20.

That is, the image processing unit 14 according to the second exampleembodiment selects a B pixel (or G pixel) as a target pixel, selects anR pixel corresponding to the target pixel, extracts R pixels havingluminance values close to a luminance value of the selected R pixel fromperipheral pixels, selects B pixels (or G pixels) corresponding to theextracted R pixels, calculates an average value Ave of luminance valuesof the selected B pixels (or G pixels) and an output value Output, andreplaces a luminance value of the target pixel with the output valueOutput so that the noise can be removed.

As described above, the image processing apparatus and the imageprocessing method according to the second example embodiment canacquiring an image excellent in color reproducibility and colorresolution by appropriately removing the noise similarly to thoseaccording to the first embodiment.

Various modifications can be made in the image processing apparatus orthe image processing method according to the first embodiment. Forexample, although the imaging element 22 is formed of two layers ofimaging pixels in the second embodiment, the imaging element 22 may beformed of three or more layers of imaging pixels.

Further, when the subject is a daytime sky, the B pixels may be used tocorrect the target pixels, i.e., the R or G pixels. When the subject isa daytime forest, the G pixels may be used to correct the target pixels,i.e., the R or B pixels.

As described above, the image processing apparatus according to thesecond example embodiment includes an image processing unit 14 thatinputs, from an imaging element 22 in which first imaging pixels havinga low SNR and second imaging pixels having a high SNR are stacked, afirst captured image by the first imaging pixels and a second capturedimage by the second imaging pixels when the first imaging pixels and thesecond imaging pixels perform imaging simultaneously, selects a targetpixel from the first captured image, extracts pixels having luminancevalues, which are close to a luminance value of a pixel corresponding tothe target pixel in an interpolated image of the second captured image,from the interpolated image, selects pixels corresponding to theextracted pixels from the first captured image, and corrects a luminancevalue of the target pixel based on luminance values of the selectedpixels.

Accordingly, it is possible to acquire an image excellent in colorreproducibility and color resolution by appropriately removing noisefrom a noisy image captured in the dark.

Further, in the image processing apparatus according to the secondexample embodiment, in the imaging element 22, the second imaging pixelsmay be in an upper layer than the first imaging pixels.

Accordingly, it is possible to maintain the SNR of the second imagingpixels having the high SNR.

Third Embodiment

The image processing apparatus and the image processing method accordingto the first and second example embodiments remove the noise from theRGB image captured by the imaging element having the RGB imaging pixels.However, an image processing apparatus and an image processing methodaccording to the third example embodiment remove noise from an RGB imageby using an imaging element that further includes a near-infrared (NIR)imaging pixels having a SNR that is higher than that of the RGB imagingpixels in the dark or in a dark part.

First, a reason for using the NIR imaging pixels in the image processingapparatus and the image processing method according to the third exampleembodiment is briefly described.

FIG. 9 is a diagram showing a relationship between a color temperature,a wavelength and a radiant energy. In FIG. 9, the horizontal axisrepresents the wavelength (nm), the vertical axis represents the radiantenergy (W/nm), the solid line represents a sunlight radiation spectrum(color temperature: 5762K), and the broken line represents a night viewradiation spectrum (color temperature: 3200K).

The radiant energy is large at each wavelength in sunlight with the highcolor temperature, but small at each wavelength in a night scene withthe low color temperature, and barely has its peak in the near-infraredray. Therefore, in the dark, imaging is performed in the near-infraredwavelength band in addition to the visible light wavelength band so thatNIR pixels having the higher SNR than that in the visible lightwavelength band can be obtained.

FIG. 10 is a diagram for schematically explaining the image processingmethod according to the third example embodiment. In FIG. 10, a leftfigure shows the image processing method according to the first andsecond example embodiments, and a right figure shows the imageprocessing method according to the third embodiment. In the first andsecond embodiments, an R signal having the high SNR in the dark or inthe dark part is used to remove noise from G and B signals having thelow SNR in the dark or in the dark part. However, in the thirdembodiment, an NIR signal having the SNR that is higher than that of theR signal in the dark or in the dark part is used to remove the noisefrom the R, G, and B signals.

Hereinafter, the image processing apparatus and the image processingmethod according to the third example embodiment are described withreference to the drawings.

The image processing apparatus according to the third embodiment is animaging apparatus, and particularly functions as an image processingunit and removes the noise from the captured image. The schematicconfiguration of the imaging apparatus may be the same as that accordingto the first example embodiment shown in FIG. 2, and illustration anddescription thereof are omitted herein.

FIG. 11 is a diagram for explaining a schematic configuration of animaging element 32 according to the third example embodiment. FIG. 11shows an enlarged view of a part of an imaging surface in the imagingelement 32.

In the imaging element 32, odd-numbered or even-numbered G imagingpixels in the Bayer array are replaced with NIR imaging pixels, and theNIR imaging pixels and RGB imaging pixels are arranged on the samelayer. The NIR imaging pixel is, for example, an imaging pixel includinga photoelectric conversion unit that performs imaging in a near-infraredwavelength band of 800 to 900 nm.

FIG. 12 is a flowchart showing an image processing method according tothe third example embodiment, and FIG. 13 is a diagram for explaining animage processing method according to the third embodiment.

Referring to FIGS. 12 and 13, in the image processing method accordingto the third embodiment, in operation S210 when the shutter of theimaging apparatus 1 is pressed, the NIR imaging pixels and RGB imagingpixels arranged on the same layer of the imaging element 32 performimaging simultaneously. In operation S220, the image processing unit 14generates an NIR interpolated image using NIR pixels in a capturedimage.

Next, in operation S230 of FIG. 12, which corresponds to operation S21of FIG. 13, the image processing unit 14 selects an R, G, or B pixel asa target pixel from the captured image, and in operation S240, whichcorresponds to operation S22 of FIG. 13, selects an NIR pixel from theNIR interpolated image as the corresponding pixel. In operation S250,which corresponds to operation S23 of FIG. 13, the image processing unit14 sets a search luminance range. In operation S260, which correspondsto operation S24 of FIG. 13, the image processing unit 14 extractsperipheral NIR pixels having luminance values close to a luminance valueof the corresponding pixel from the NIR interpolated image.

Next, in operation S270, which corresponds to operation S25 of FIG. 13,the image processing unit 14 selects R, G, or B pixels from the capturedimage as corresponding pixels of the extracted NIR pixels. Thereafter,in operations S280 and S290, which correspond to operations S26 and 27of FIG. 13, respectively, the image processing unit 14 calculates areplacement value and corrects the target pixel to remove. In operationS300, the noise and the corrected image is displayed and stored.

As described above, the image processing apparatus and the imageprocessing method according to the third example embodiment canacquiring an image excellent in color reproducibility and colorresolution by appropriately removing the noise similarly to thoseaccording to the first and second example embodiments.

Modification of Third Embodiment

Various modifications can be made in the image processing apparatus orthe image processing method according to the second example embodiment.For example, although the NIR imaging pixels and the RGB imaging pixelsare arranged on the same layer as the imaging element 32 in the thirdexample embodiment, the NIR imaging pixels and the RGB imaging pixelsmay be stacked in the imaging element.

FIG. 14 is a diagram for explaining a schematic configuration of animaging element 42 according to a modification of the third exampleembodiment. FIG. 14 shows a cross section when a pixel row (or pixelcolumn) of the imaging element 42 and a pixel row (or pixel column)adjacent thereto are viewed from a direction perpendicular to a stackingdirection of the imaging element 42.

The imaging element 42 has a stacked structure using a well-knowntechnique, for example, in which a lower layer is formed of asemiconductor material such as a silicon substrate and an upper layer isformed of a semiconductor material or an organic film. In the lowerlayer, R, G and B imaging pixels having the low SNR in the dark or inthe dark part are arranged in the Bayer array. In the upper layer, theNIR imaging pixels having the high SNR in the dark or in the dark partare arranged on an entire surface so as to correspond to the R, G and Bimaging pixels. That is, in the upper layer, all the imaging pixels arethe NIR imaging pixels.

In a captured image captured by such an imaging sensor 42, in order toobtain an NIR pixel having the low SNR and an R, G or B pixel having thehigh SNR and at the same position (coordinates) of the image and removenoise of the R, G or B pixel, the same processing as the imageprocessing method according to the first embodiment may be performed. Inthis case, since the NIR pixels of all the pixels are obtained, themodification of the third embodiment may not generate an NIRinterpolated image.

Although the imaging element 42 is formed of two layers of imagingpixels in the modification of the third embodiment, the imaging element42 may be formed of three or more layers of imaging pixels. Further, theimaging pixels having the high SNR may not be in an upper layer.

As described above, in the image processing apparatus according to thethird example embodiment, the first imaging pixels may perform theimaging in a red wavelength band, a green wavelength band, or a bluewavelength band, and the second imaging pixels may perform the imagingin a near-infrared wavelength band.

Accordingly, it is possible to correct the RGB pixels and remove thenoise using the NIR pixels having the high SNR.

In some example embodiments, the image processing unit 14 may sense aluminance of a scene and selectively switch between operating a firstmode in which the image processing apparatus 14 performs the imageprocessing method of FIG. 1 and a second mode in which the imageprocessing apparatus performs the image processing methods of one of thefirst to third example embodiments based on the sensed luminance. Forexample, in the first mode, the image processing unit 14 may search forpixels having the same color as the target pixel, and calculate theaverage value of these pixels having the same color as the target pixel,and, in the second mode, the image processing unit 14 may select G pixelor B pixel having a low SNR in the dark or in a dark part as a targetpixel, extract peripheral R pixels based on a luminance value of an Rpixel corresponding to the target pixel and having a high SNR in thedark or in the dark part, select G pixels or B pixels corresponding tothe extracted R pixels, and correct the target pixel based on luminancevalues of the selected G pixels or B pixels.

While example embodiments have been described in connection withreference to some example embodiments, it is to be understood that theexample embodiments is not limited to the disclosed example embodiments.On the contrary, it is intended to cover various modifications andequivalent arrangements included within the spirit and scope of theappended claims.

What is claimed is:
 1. An image processing apparatus comprising: processing circuitry configured to, receive a first captured image and a second captured image simultaneously captured via first imaging pixels and second imaging pixels, respectively, that are one of stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels, select a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel in the second captured image or an interpolated image of the second captured image, extract pixels from the second captured image or the interpolated image of the second captured image based on a luminance value of the corresponding pixel to generate extracted pixels, select pixels corresponding to the extracted pixels from the first captured image as selected pixels, and correct a luminance value of the target pixel based on luminance values of the selected pixels.
 2. The image processing apparatus of claim 1, wherein the processing circuitry is configured to generate the extracted pixels such that luminance values of the extracted pixels fall within a set luminance range of the luminance value of the corresponding pixel.
 3. The image processing apparatus of claim 2, wherein the set luminance range is based on a magnitude of the luminance value of the corresponding pixel.
 4. The image processing apparatus of claim 1, wherein the processing circuitry is configured to generate the extracted pixels by, extracting the pixels from a set number of pixels in a vertical direction and a horizontal direction around the corresponding pixel, based on the luminance value of the corresponding pixel.
 5. The image processing apparatus of claim 1, wherein the processing circuitry is configured to calculate an average value of the luminance values of the selected pixels, and correct the luminance value of the target pixel by a value determined based on the luminance value of the target pixel, the average value, and a set gain.
 6. The image processing apparatus of claim 1, wherein the first imaging pixels are configured to perform imaging in a red wavelength band, a green wavelength band, or a blue wavelength band, and the second imaging pixels are configured to perform imaging in a near-infrared wavelength band.
 7. The image processing apparatus of claim 1, wherein the first imaging pixels are configured to perform imaging in a green wavelength band or a blue wavelength band, and the second imaging pixels are configured to perform imaging in a red wavelength band.
 8. The image processing apparatus of claim 1, wherein the second imaging pixels are in an upper layer of the imaging element as compared to the first imaging pixels.
 9. The image processing apparatus of claim 1, wherein the processing circuitry is further configured to select pixels corresponding to the extracted pixels from an interpolated image of the first captured image.
 10. An image processing apparatus comprising: processing circuitry configured to, receive a first captured image and a second captured image simultaneously captured via first imaging pixels and second imaging pixels, respectively, that are one of stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels, select a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel in the second captured image or an interpolated image of the second captured image, select, as selected pixels, pixels from the first captured image based on a luminance value of the corresponding pixel in the second captured image or the interpolated image, and correct a luminance value of the target pixel based on luminance values of the selected pixels.
 11. The image processing apparatus of claim 10, wherein the processing circuitry is configured to select the selected pixels such that luminance values of pixels corresponding to the selected pixels in the second captured image or the interpolated image fall within a set luminance range of the luminance value of the corresponding pixel.
 12. The image processing apparatus of claim 11, wherein the set luminance range is based on a magnitude of the luminance value of the corresponding pixel.
 13. The image processing apparatus of claim 10, wherein the processing circuitry is configured to extract the pixels corresponding to the selected pixels by, extracting the pixels from a set number of pixels in a vertical direction and a horizontal direction around the corresponding pixel, based on the luminance value of the corresponding pixel.
 14. The image processing apparatus of claim 10, wherein the processing circuitry is configured to calculate an average value of the luminance values of the selected pixels, and correct the luminance value of the target pixel by a value determined based on the luminance value of the target pixel, the average value, and a set gain.
 15. The image processing apparatus of claim 10, wherein the first imaging pixels are configured to perform imaging in a red wavelength band, a green wavelength band, or a blue wavelength band, and the second imaging pixels are configured to perform imaging in a near-infrared wavelength band.
 16. The image processing apparatus of claim 10, wherein the first imaging pixels are configured to perform imaging in a green wavelength band or a blue wavelength band, and the second imaging pixels are configured to perform imaging in a red wavelength band.
 17. The image processing apparatus of claim 10, wherein the second imaging pixels are in an upper layer of the imaging element as compared to the first imaging pixels.
 18. The image processing apparatus of claim 10, wherein the processing circuitry is further configured to select pixels from an interpolated image of the first captured image based on the luminance value of the corresponding pixel.
 19. A non-transitory computer readable medium storing image processing program that, when executed by a computing device, configures the computing device to: receive a first captured image and a second captured image by simultaneously captured via first imaging pixels and second imaging pixels, respectively, that are one of stacked or are arranged on a same layer of an imaging element, the second imaging pixels having a higher signal-to-noise ratio (SNR) than the first imaging pixels; select a target pixel from the first captured image, the target pixel in the first captured image having a corresponding pixel corresponding thereto in the second captured image or an interpolated image of the second captured image; select, as selected pixels, pixels from the first captured image based on a luminance value of the corresponding pixel in the second captured image or the interpolated image; and correct a luminance value of the target pixel based on luminance values of the selected pixels.
 20. The non-transitory computer readable medium of claim 19, wherein the selecting pixels includes: extracting pixels from the second captured image or the interpolated image of the second captured image based on the luminance value of the corresponding pixel to generate extracted pixels; and selecting the selected pixels corresponding to the extracted pixels from the first captured image. 